Digital capture of videos of scenes is a useful and commonly-practiced technique. Videos are typically captured as a series of individual frames at a selected frame rate, e.g. 24 frames per second (fps), and each frame is captured by accumulating light for a selected exposure time within the available frame time, e.g. 41.7 millisec (= 1/24 sec.). Each frame is a two-dimensional array of individual pixels.
Scenes, and thus videos of scenes, can contain global motion or local motion. Global motion refers to relative motion between the image capture device and the scene being imaged, such as when a camera is panning across a scene. Local motion refers to motion of objects within the scene, such as a ball being thrown. When a video contains either global or local motion, and the motion that occurs during the exposure time of a frame causes the light from an object to spread across more than one pixel, image quality can be degraded in the form of blurring and smearing of the image. For example, a five megapixel (5 Mp) camera which has a lens with a 50 degree field of view operating at 30 fps (33 ms exposure time) produces blur of more than two pixels when capturing video of an object located 15 feet from the camera that is moving faster than 4 inches/sec (the equivalent of a very slow wave). It is therefore desirable to reduce blur of rapidly moving objects in video frames to improve video image quality.
Typically higher quality video such as high definition (HD) video (720p, 1080i or 1080p) is captured at a higher frame rate, e.g. 30 fps or 60 fps, to reduce the blurring associated with motion during capture. However, when rapid motion is present in the scene, such as a ball being thrown in a sporting event, the image of the ball can be noticeably blurred even when captured at 60 fps. Very fast frame rates can be used to reduce blur and improve video image quality of rapidly moving objects. However, as the frame rate is increased, the amount of image data associated with the video increases proportionately, which can result in data rates too high for data storage, image processing or data transmission bandwidth in imaging systems such as a consumer video camera, a digital camera or a cell phone camera. As a result, consumer imaging devices are typically limited to frame rates of 30 fps or 60 fps at 720p or 1080p resolutions.
Video compression techniques can reduce data transmission bandwidth and data storage requirements by detecting changes such as mean absolute differences between frames and avoiding the transmission of duplicate image data for multiple frames when the scene is not changing. U.S. Pat. Nos. 6,931,065 and 5,969,764 ('764) include motion estimation in their respective compression techniques. As such, the technique described in the '764 patent reduces the data transmission rate for regions of the scene that are not changing but keeps the original capture data transmission rate for areas where motion is present. However, this method does not reduce data rate or blur in the video frames as captured.
Motion detection between images, such as between the frames in a video, is described in U.S. Pat. Nos. 7,403,640 ('640), 7,385,626 and 6,931,065. The '640 patent is further described in Zhang, B. (2003), “Regression Clustering,” IEEE ICDM'03 Proceedings, 0-7695-1978-4/03. These techniques assess the speed of motion that is present within the scene and identify regions in the video frames where rapid motion is present such as a person running or a ball being thrown. The motion assessment can result in a motion map or a series of motion vectors for individual pixels or groups (or regions) of pixels within the images with speed data or speed and direction data. This motion assessment information can be used to reduce the data transmission rate by reusing image content for an object even as it moves. However, these schemes have no effect on blur in the captured video.
U.S. Pat. No. 5,389,965 describes a variable frame rate system for video communication. This system permits the user to select the frame rate used to deliver the desired image quality in a mobile communication environment where data transmission bit rates are limited. Slow frame rates are thus used to deliver higher resolution images at the expense of jerky motion. Faster frame rates deliver smoother motion with lower resolution images. This approach does not address the need for improved capture of rapidly moving objects.
In addition, increasing the frame rate can reduce the exposure time for each frame to the point that noise becomes a problem in the frames in low light situations. As the frame rate is increased, the available exposure time for each frame is correspondingly decreased and the noise present in each frame increases. Although the noise is not as noticeable to a viewer in portions of frames which contain motion, due to frequency masking in the human visual system, noise can be very noticeable in portions of frames which contain no motion or slow motion. As a result, it is desirable to avoid high noise levels in regions of the frame in which slow motion or no motion is present.
Consequently, there exists a need for a faster frame rate capture of video for rapid motion in a way that does not substantially increase the amount of image data associated with the video or significantly increase the noise level in static regions of the image.